test: add live end-to-end validation harness - #75
Open
JNK234 wants to merge 1 commit into
Open
Conversation
Test procedures that make real API calls through the installed extension, covering behaviour that unit tests cannot reach because it depends on what a provider actually returns. - T1 basic chat round trip - T2 thinking-tag leakage: a plain llm:chat never sends reasoning_format, so a provider defaulting to "raw" would return reasoning inline in <think> tags with content copied verbatim. Only a live call settles this. - T3 thinking enabled: reasoning must arrive separately, not duplicated - T4 llm:compile-error - T5 per-agent history isolation and clear-history - T6 structured output: asserts fields arrive as NetLogo TYPES, not text — a provider can return well-formed JSON and still fail if the conversion drops types, so is-number?/is-boolean? are checked alongside the enum constraint, and a missing key must raise rather than default Configs ship for ollama (no key) plus groq and anthropic as .example templates; the key-bearing copies are gitignored. Verified: 22/22 pass against live Groq (openai/gpt-oss-20b) via headless BehaviorSpace, real HTTP, no stubs. Headless note: NetLogo 7.0.3 only runs an experiment embedded in the .nlogox. An external --setup-file is accepted and then silently ignored, aborting with "head of empty list" before setup runs. The embedded block requires sequentialRunOrder, timeLimit as an attribute, and metrics wrapped in <metrics>.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The unit suite is deterministic by design — it asserts request payloads without a live call, so CI needs no keys. That proves the plumbing, not that a provider accepts the request or that its response parses.
Today's review cycle showed the gap is not theoretical. Two findings derived carefully from vendor documentation were each settled in the opposite direction by one real call:
reasoning_format: "raw", so<think>tags leak into visible content — rated HIGHgpt-oss-20b. Returns cleancontentplus a separatereasoningfield.reasoning_effortmismatch — rated MEDIUMxhighpassed extension validation and returned HTTP 400.What
demos/e2e-tests/— six test procedures making real API calls through the installed extension, with a chooser for the provider under test.llm:compile-error(#52)clear-historyT6 asserts types, not just shape. A provider can return well-formed JSON and still fail if the conversion drops types, so
is-number?andis-boolean?are checked alongside the enum constraint, and a missing key must raise rather than default silently.Verification
22/22 pass against live Groq (
openai/gpt-oss-20b), headless, real HTTP, no stubs:config-ollama.txtneeds no key, so the suite runs offline against a local model too.Keys
config-groq.txtandconfig-anthropic.txtare gitignored and ship as.exampletemplates. Onlyconfig-ollama.txt, which needs no key, is tracked.Headless finding worth recording
NetLogo 7.0.3 only runs an experiment embedded in the
.nlogox. An external--setup-fileis accepted and then silently ignored, aborting withNoSuchElementException: head of empty listbeforesetupruns — indistinguishable from the benign 0-step abort. The embedded block additionally requiressequentialRunOrder,timeLimitas an attribute rather than a child node, and metrics wrapped in<metrics>.This cost real debugging time and is not documented anywhere in the repo.